Gain Control for a Hearing Aid with a Facial Movement Detector

ABSTRACT

Methods, systems and devices are provided for controlling a hearing aid. Embodiments include receiving an input audio signal from a microphone. Also, a facial movement indication may be received from a facial movement detector measured contemporaneously with the input audio signal. Additionally, whether a first movement pattern associated with the wearer speaking is a movement match to the facial movement indication may be determined. In this way, a first gain profile may be applied to the input audio signal for generating an augmented audio segment in response to determining the first movement pattern is the movement match to the facial movement indication. Also, a second gain profile may be applied to the input audio signal for generating the augmented audio segment in response to determining the first movement pattern does not match the facial movement indication. Thus, the augmented audio segment may be output.

BACKGROUND

Hearing loss is generally caused by sensorineural hearing loss,conductive hearing loss or a combination of the two. Sensorineuralhearing loss, occurs when there is damage to the inner ear (i.e., thecochlea) or to the nerve pathways from the inner ear to the brain.Sensorineural hearing loss is not generally correctable using medicinesor surgery and is unfortunately the most common type of permanenthearing loss. Conductive hearing loss occurs when sound is not conductedefficiently through the outer ear canal to the eardrum and the tinybones (ossicles) of the middle ear. Conductive hearing loss usuallycauses a reduction in sound level or the ability to hear faint sounds.Unlike sensorineural, conductive hearing loss can often be correctedmedically and/or surgically.

Sensorineural hearing loss is the most common type of hearing loss amongadults (occurring in 80 percent of adult hearing loss cases). Althoughsensorineural hearing loss is not often medically or surgicallytreatable, the use of hearing aids often helps. However, contemporaryhearing aids do not work very well at helping the wearer hear soundswhen the wearer himself is speaking, since the speaker's own sounds tendto get over amplified. The over amplification is due in part to naturalphenomenon that occurs when the ear canal is blocked (particularly by ahearing aid). People with normal hearing can simulate this phenomenon byplacing a finger in an ear and listening to their own speech. To addressthis issue, one approach is to providing venting holes in the part of ahearing aid that gets inserted in the ear canal. Another approach lowersthe output gain of the hearing aid in lower frequencies, whichcorrespond to the added sound heard by the wearer when he or she isspeaking. However, these approaches often reduce the effectiveness ofthe hearing aid when the user is not talking.

SUMMARY

The various embodiments described herein include methods, systems anddevices for controlling the output of a hearing aid worn by a wearerbased on whether the wearer is speaking as determined by facial movementsensors. The audio output of the hearing aid may be controlled toamplify an audio signal received by a microphone based on a facialmovement indication received from a facial movement detector measuredcontemporaneously with the input audio signal. A first gain profile maybe applied to the input audio signal for adjusting or generating anaugmented audio segment in response to determining that facial movementsmatch a stored facial movement pattern correlated to the wearerspeaking. Also, a second gain profile may be applied to the input audiosignal for generating the augmented audio segment in response todetermining that facial movements do not match a stored facial movementpattern correlated to the wearer speaking. In an embodiment, a userselection input may be received indicating whether the first gainprofile should be applied, and the first gain profile may be applied tothe input audio signal when the selection input indicates the first gainprofile should be applied and the second gain profile may be applied tothe input audio signal when the selection input indicates that the firstgain profile should not be applied.

Further embodiments may include a hearing aid having a processorconfigured with processor-executable software instructions to performvarious operations corresponding to the methods discussed above.

Further embodiments may include a hearing aid having various means forperforming functions corresponding to the method operations discussedabove.

Further embodiments may include a non-transitory processor-readablestorage medium having stored thereon processor-executable instructionsconfigured to cause a processor to perform various operationscorresponding to the method operations discussed above.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are presented to aid in the description ofembodiments of the disclosure and are provided solely for illustrationof the embodiments and not limitation thereof

FIG. 1 illustrates a partial cross-sectional view of a human auditorysystem and a hearing aid suitable for use in the various embodiments.

FIG. 2 illustrates a schematic block diagram of a hearing aid suitablefor the various embodiments.

FIG. 3 illustrates comparative graphs of data received from a microphoneand an EMG sensor.

FIG. 4A illustrates comparative audiogram profile representing level ofhearing loss based on a subject hearing “own speech” versus hearingeverything else.

FIG. 4B illustrates comparative gain profiles corresponding to theaudiogram profiles of FIG. 4A.

FIG. 5 illustrates a graph with gain profiles for varied frequencybands, suitable for use in the various embodiments.

FIG. 6 illustrates a process flow diagram suitable for use in thevarious embodiments.

FIG. 7A illustrates a side view of a hearing aid suitable for thevarious embodiments worn by a wearer.

FIG. 7B illustrates a perspective view of the hearing aid of FIG. 7Aalone.

DETAILED DESCRIPTION

The various embodiments will be described in detail with reference tothe accompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.References made to particular examples and implementations are forillustrative purposes, and are not intended to limit the scope of thedisclosure or the claims. Alternate embodiments may be devised withoutdeparting from the scope of the disclosure. Additionally, well-knownelements of the disclosure will not be described in detail or will beomitted so as not to obscure the relevant details of the disclosure.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any implementation described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other implementations. Additionally, use of the words,“first,” “second,” “third,” “primary,” “secondary,” “tertiary” orsimilar verbiage is intended herein for clarity purposes to distinguishvarious described elements and is not intended to limit the invention toa particular order or hierarchy of elements.

The various embodiments relate to improving the output sound of hearingaids by applying different gain profiles to an output of a hearing aidbased on whether the user is speaking or not. The hearing aid may useone or more facial movement detectors, such as an electromyographysensor (EMG sensor), to detect facial movements that correspond to theuser speaking. Also, one or more microphones may be used to detectwhether the input audio signals detected by the hearing aid correspondto recognizable voice patterns of the wearer. Based on determinationsmade from muscle activity indicated from the facial movement detectorand/or the input audio signal from the microphone, an appropriate gainprofile may be applied to the input audio signal. In this way, thehearing aid output sound is adjusted based on the determinations madecorresponding to one or more indications the wearer may be speaking.

FIG. 1 illustrates a partial cross-sectional view of a representativehuman auditory system 5 in conjunction with a hearing aid 100 suitablefor use in the various embodiments. Each ear that is part of the typicalhuman auditory system 5 includes the pinna 10 (i.e., the externallyprotruding parts of the ear, including the lobes) an ear canal 15, whichare part of the outer ear. Damage to portions of the outer ear and/orproblems with how the ear canal 15 conducts sounds may result inconductive hearing loss. The ear canal 15 extends from the outermostparts of the ear to the tympanic membrane 20 (also referred to as theear drum), which forms part of the middle ear that includes ossicles 25and joins to the Eustachian tube 30. The inner ear includes semicircularcanals 35, a cochlea 40 and the auditory nerve 45. Damage to the innerear is common with sensorineural hearing loss.

Also shown in FIG. 1 is a hearing aid 100, which includes a microphone110, an ear hook 115, a facial movement detector 120 and a speaker 130.The hearing aid 100 may be formed, enhanced and/or sized to conform tothe shape of a portion of the wearer's ear. The microphone 110 receivesan input audio signal A₁, cause by sounds generated in the surroundingenvironment. Also, the microphone 110 may include more than onemicrophone for picking up sounds from different directions, such asthose sounds coming from the wearer, as compared to sounds from anothersource. Elements such as the ear hook 115 are optional. The facialmovement detector 120 may be configured to engage a portion of thewearer's skin suitable for detecting facial movements, particularlymuscle movements, associated with speaking. In this way, the facialmovement detector 120 may be configured to detect signals M₁ generatedby facial muscles when the wearer is speaking. In this way, facialmovements not associated with the wearer speaking, such as eating,drinking, chewing or even certain silent facial expressions, may bedistinguished from facial movements associated with speech. The speaker130 may be configured to output the augmented sound A₂ generated by thehearing aid 100 in accordance with the various embodiments. The hearingaid 100 may further include one or more processors (not shown)configured with processor-executable instructions to perform operationsas disclosed herein. The processor(s) may be coupled to the microphone110, the facial movement detector 120 and the speaker 130, in order toreceive inputs A₁, M₁ and generates an output sound A₂ for the wearer'sauditory system 5 to more easily and clearly hear.

As used herein, the term “hearing aid” refers to an electro-acousticdevice, which typically fits in or behind a wearer's ear, and may bedesigned to amplify and modulate sound for the wearer. The hearing aid,as referred to herein adds a certain level of gain to the incoming soundto generate an augmented sound for the wearer to hear. The incomingsound may include an input audio signal, all or a portion of which maybe analyzed by a hearing aid processor. The gain or level of gain may bethe amount of sound added by the hearing aid. Gain as used herein refersto the difference between a hearing aid's input level and the outputlevel, which levels may be measured in decibels. A hearing aid may applya gain to incoming sounds (received as an input audio signal) in orderto generate a louder and/or modified output sound. An input audio signalmay be a representation of sound, such as an electrical voltage,generally having multiple frequencies. Sound refers to a mechanical wavethat is an oscillation of pressure. For humans the audible range ofsound frequencies generally ranges from 16 Hz to 20,000 Hz. Differinglevels of gain may be added to different frequencies of the input audiosignal. In this way, frequencies in which the user has difficultyhearing may have more gain applied than other frequencies. A gainprofile refers to a set of data correlating ranges of sound frequenciesto varying levels of gain.

As used herein, the terms “microphone” or “hearing aid microphone” areused interchangeably herein and refer to an input transducer of ahearing aid that picks up sound (one or more input audio signals) fromthe immediately surrounding environment and converts it into anelectrical signal, which it directs to a processor/amplifier foramplification and/or modulation.

As used herein, the term “facial movement detector” refers to a sensorcapable of detecting facial movement, particularly those facialmovements associated with a hearing aid wearer speaking. A facialmovement detector may be able to receive a facial movement indication,which is a representation of the movement of facial muscles and/or thesurface skin associated with the movements of the face. In the variousembodiments, the facial movement detector may be particularly suitedand/or situated to detect facial movement associated with speaking. Anexemplary facial movement detector in accordance with an embodiment isan electromyography (EMG) sensor. EMG is a technique for evaluating andrecording the electrical activity produced by skeletal muscles. An EMGsensor may detect signals in the form of the electrical potentialgenerated by muscle cells when these cells are electrically orneurologically activated. The signals may be analyzed to detectbiomechanics of human, such as jaw movements corresponding to a personspeaking. A facial EMG may measure facial movement activity by detectingand amplifying the tiny electrical impulses that are generated by facialmuscle fibers when they contract. Another form of facial movementdetector may include one or more conductive textile electrodes placed incontact with the skin, which may detect changes caused by muscle motion,tissue displacement and/or electrode deformation. A further facialmovement detector may be a pressure sensor configured to detect skinsurface changes, particularly at or near the wearer's jaw. Furtherstill, another microphone configured to detect sound conducted throughthe wearer's facial tissue, including bones, may be used as a facialmovement detector.

As used herein, the term “speaker” or “receiver” are usedinterchangeably herein and refer to a component of a hearing aid thatchanges electrical signals from the processor/amplifier into sound,which is generally directed into the ear of the wearer.

FIG. 2 illustrates an embodiment schematic block diagram of theelectronic components of a hearing aid 100. As above, the hearing aid100 includes microphone 210 for receiving input audio signals. Themicrophone 210 may include more than one microphone. For example, afirst microphone may be configured on the hearing aid to more readilypick up sounds from the wearer's own mouth, while a second microphonemay be configured to primarily pick up other ambient sounds. Otherwise,the microphone 210 serves the traditional function of receiving audiosignals, which may be processed through an analog/digital (AD) converter215 for digital processing of the detected/received signals. Thereceived ambient audio signal may then be processed as one or more inputaudio signals through an Input Audio Analyzer/Filter (IAAF) 220.

The IAAF 220 may be a unit that detects whether a voice is present in aninput audio signal. In particular, the IAAF 220 may be configured tospecifically detect the wearer's own voice by applying frequencyanalysis to determine one or more fundamental frequencies of thereceived electrical signal. Thus, the IAAF 220 may act as a voicedetector by comparing the electric (i.e., digitized) representation ofthe acoustic input sounds to one or more sets of frequency patternscorrelated to human speech. Particularly, as part of the setup of theIAAF 220 and the overall hearing aid 100, frequency patterns of thewearer's own speech may be stored in an operatively coupled memory forcomparison and matching to the digitized acoustic input signal.Alternatively, the presence of synchronous patterns and harmonicstructures of the sounds associated with one or more designatedlanguages, words and/or even letters may be used to identify voiceactivity. In this way, the IAAF 220 may determine whether at least aportion of the input audio signal, such as characteristics representedby an input audio signal pattern, is a match to similar characteristicsof a first voice pattern associated with speech generated by the wearer.A match of an input audio signal with a voice pattern means the twopatterns (each representing an audio signal) are substantiallyequivalent. Additionally, the IAAF 220 may serve as a filter,identifying predefined sounds, undesirable noises and/or patterns(collectively referred to as “noise”) for which the hearing aid need notapply a gain. The portion of the input audio signal identified to benoise may bypass the Gain Control Processor 250 and be sent directly tothe mixer 270 for output. In this way, those portions identified as“noise” may still be output by the hearing aid, but not necessarilyamplified or even attenuated. Otherwise, those other portions of theinput audio signal not considered noise may be forwarded to the GainControl Processor 250, along with any indication as to whether anysub-portion thereof has been identified as human speech and/or thewearer's speech. Alternatively, those portions identified as “noise” maybe attenuated by the mixer or filtered out entirely.

Additionally, the hearing aid 100 may include a facial movement detector230 for receiving facial movement indications, particularly from facialmuscles. For example, an EMG sensor that may include surface electrodesfor measuring a voltage differential, may serve as a facial movementdetector 230. A facial movement detector 230 may be located in directcontact with the hearing aid wearer's skin. For example, the facialmovement detector 230 may be positioned on an external portion of thehearing aid 100 in contact with facial regions whose movement isassociated with speaking. The facial movement detector 230 may includemore than one facial movement detector in order to detect/differentiatepatterns of facial movement and/or to provide redundancies to ensuremovement is detected. For example, a first facial movement detector maybe disposed on a first part of the hearing aid, while a second facialmovement detector may be disposed remote from the first facial movementdetector on a second part of the hearing aid or even remote from themain hearing aid body. Otherwise, the facial movement detector 230serves to receive facial movement indications, which may be processedthrough an analog/digital (AD) converter 235 for digital processing ofthe signals representing those facial movement indications. The receivedfacial movement indications may then be processed as one or more inputsignals through a Muscle Activity Analyzer (MAA) 240.

The MAA 240 may be a unit that amplifies, decomposes and processes thereceived facial movement indications. For example, measured EMG signalsmay be decomposed and processed into their constituent motor unit actionpotentials, some of which may have particular characteristics associatedwith muscle movements associated with speech. Additionally, the MAA 240may analyze those processed facial movement indications, by isolatingthose portions relevant to recognizing when the hearing aid wearer isspeaking. In particular, the MAA 240 may act as a speech detector bybeing configured to specifically detect which jaw muscle movements areassociated with speech. The MAA 240 may compare the electric (i.e.,digitized) representation of facial movements (i.e., a facial movementindication) to one or more sets of patterns generally correlated tofacial movements during human speech. A more customized analysis maycompare the detected patterns to previously recorded movements of aparticular wearer while speaking. As part of the setup of the MAA 240and the overall hearing aid 100, facial movement patterns of the wearer,while speaking, may be stored in an operatively coupled memory forcomparison and matching to the received facial movement indication.Facial movement patterns may be measured by sensors, converted into asignal, which will have its own representative pattern of the actualfacial movement pattern, and stored and/or analyzed. Alternatively, thepresence of generic patterns associated with human speech may be used toidentify movement patterns indicative of the wearer speaking. Thedetermination as to whether the facial movement indication matches(i.e., is a movement match) one or more predefined (stored) movementpatterns associated with the wearer speaking may be forwarded to theGain Control Processor 250. A match of a facial movement indication(based on its representative pattern), received from a facial movementdetector, to a stored facial movement pattern means the two patterns(each representing facial movement associated with a wearer speaking)are substantially equivalent.

The Gain Control Processor (GCP) 250 may be a processor capable ofproperly sorting and/or analyzing the signals from the IAAF 220 and/orthe MAA 240. In an embodiment, the IAAF 220 and/or the MAA 240 maydeliver raw or only partially processed signals to the GCP 250, in whichcase the GCP may further process those signals. In fact, many or most ofthe functions of the IAAF 220 and/or the MAA 240 may be performed by theGCP 250 and particularly a Wearer Speech Detection Unit (WSPU) 260 thatmay be part of the overall GCP 250. Thus, the WSPU 260 may receive boththe input audio signal and the facial movement indication in order todetermine whether the facial movement indication is a movement match toa first movement pattern associated with the wearer speaking.Additionally, the WSPU 260 may determine whether at least a portion ofthe input audio signal is an audio match to a first voice patternassociated with speech generated by the wearer. Based upon the receivedinputs, the WSDU 260 may determine whether to apply a first gain profileP₁ or a second gain profile P₂ to an entire input audio signal orportions thereof. The first gain profile P₁ may be applied when it isdetermined that the wearer is speaking; while the second gain profile P₂may be applied when it is determined that the wearer is not speaking. Asdescribed further below, each of the first gain profile P₁ or a secondgain profile P₂ may be further broken down across various frequencyranges. In this way the gain applied to incoming sound is further varieddepending upon what frequency ranges are present.

The hearing aid 100 may further include a mixer 270 for combining andchanging the output level of the original input audio signal based uponthe applied gain profiles, if applicable. Additional gain may be appliedby the mixer 270 based on detected levels of noise received from theIAAF 220. In this way, all or portions of the input audio signal may beenhanced for outputting an augmented audio segment suitable to the needsof a hearing aid wearer. The augmented audio segment may include variousportions and/or frequencies that have been enhanced in varied ways basedon the appropriate gain profiles applied. In this way, the augmentedaudio segment may include one distinct portion of the original inputaudio signal that has been changed differently than another distinctportion by have different gain profiles applied thereto. The outputsignal generated by the mixer 270 may be processed through adigital-to-analog converter (DA) 275 for converting a digitized signalto an analog output signal, if appropriate. In this way, the augmentedaudio segment may be output to the wearer of the hearing aid from thespeaker 280.

FIG. 3 illustrates representative patterns of processed inputs from themicrophone 210 and the facial movement detector 230 measured over aperiod of time. The upper graph particularly represents the processedinput audio signal detected by the microphone 210. The lower graphrepresents the processed facial movement indication detected by thefacial movement detector 230, which in this illustration is from an EMGsensor. The upper and lower graphs run contemporaneously, meaning theyrepresent separate inputs occurring in the same period of time. In thisway, the patterns or portions of the patterns may be correlated in time,such that certain portions of the input audio signal may be linked tocorresponding portions of the facial movement indications.

The input audio signal includes a measureable noise level S_(x), as wellas a number of patterns that may be analyzed and compared to knownpatterns associated with speech. A voice pattern as used herein refersto an arrangement or sequence within the representation of an inputaudio signal that is discernible and may be compared to known patternsof sound associated with human speech or a particular individual'sspeech. In fact, the illustrated input audio signal includes twoportions V₁, V₂ that match voice patterns associated with speech. Theinput audio signal need not include any patterns that match voicepatterns, but if it does preferably it may be identified and determinedto be a match. It should be understood that each of the two portions V₁,V₂ may individually be considered a separate recognizable voice pattern.Although the first voice pattern V₁ and the second voice pattern V₂ areshown as being spaced apart, they may be consecutive and/or in reverseorder (i.e., V₂ before V₁). Numerous other voice patterns that may beassociated with speech may also be available for comparison of furtherinput audio signals. As noted above, the two voice patterns V₁, V₂ maybe generic voice patterns associated with human speech. Alternatively,matching patterns may be limited to voice patterns specificallycorrelated to a particular hearing aid wearer's speech. Regardless, thispattern matching illustrates a means of determining whether at least aportion of an input audio signal matches a first voice patternassociated with speech generated by the wearer.

The lower graph also includes patterns that may be recognizable, whichas with the input audio signal may be analyzed and compared to knownmovement patterns associated with speech. A movement pattern as usedherein refers to an arrangement or sequence within the representation ofa facial movement indication that is discernible and may be compared toknown patterns of movement associated with human speech or a particularindividual's speech. The illustrated facial movement indication includesa portion F₁ that is a match to a movement pattern associated withspeaking (i.e., a movement match). It should be understood that thefacial movement indication may not include a portion that matches anymovement pattern associated with speaking. Also, the facial movementindication may include more than one portion that matches a speaker'smovement pattern. Preferably, the matching movement pattern F₁ is notjust associated with any human speaking, but more specificallyassociated with the wearer of the hearing aid speaking. Consequently,this further pattern matching illustrates a means of determining whetherat least a portion of a received facial movement indication is amovement match to a first movement pattern associated with speechgenerated by the wearer.

Thus, the GCP 250 and/or the WSPU 260 receiving the processed orsemi-processed information regarding the input audio signal and thefacial movement indication may determine what gain profile to apply forgenerating an output sound that is more readily heard/recognized by thewearer. If based on the analyzed input it is determined the wearer isspeaking, then a first gain profile may be applied to the input audiosignal. However, if based on the analyzed input it is determined thewearer is not speaking, then a second gain profile may be applied to theinput audio signal. In an embodiment, a first gain profile may beapplied to the input audio signal for generating an augmented audiosegment in response to only determining the facial movement indication,contemporaneously measured with that input audio signal, is a movementmatch to the first movement pattern F₁. Alternatively, applying thefirst gain profile may be further in response to determining the inputaudio signal is an audio match to the first voice pattern. This aspectis shown in FIG. 3, where only a portion of the matching movementpattern F₁ of the facial movement indication occurs contemporaneouslywith the second matching voice pattern V₂. Accordingly, a first gainprofile may be applied to that portion of the input audio signaloccurring during the overlapping segment of time T₁ and a second gainprofile may be applied to the portions of the input audio signaloccurring outside the overlapping segment of time T₁.

The set or sets of data used to develop an appropriate gain profile fora hearing aid is/are generally derived from an audiogram determined fromtesting conducted on the person needing the hearing aid. FIG. 4Aillustrates an audiogram showing separate hearing threshold levelsacross various frequencies. One audiogram profile (indicated as “OwnSpeech”) represents the subject's hearing threshold levels when hearinghimself speak. The other audiogram profile (indicated as “Other”)represents the subject's hearing threshold levels when hearing soundsother than his own speech, including hearing others speak. In thisexample, both profiles include varying degrees of significant hearingloss in the middle frequencies and even more hearing loss in the higherfrequencies. Thus, this individual may benefit from having acorrespondingly higher level of gain applied to those middle and higherlevel frequency portions of an input audio signal. Additionally, inaccordance with an embodiment this individual may further benefit afirst gain profile applied when he is listening to himself speak and asecond gain profile when he is not speaking. FIG. 4B illustrates first(1^(st)) and second (2^(nd)) gain profiles, respectively correspondingto the “own speech” and “other” scenarios from FIG. 4A. Each of the gainprofiles includes a set of gain values corresponding to multiplefrequencies of the input audio signal. In this way, the “own speech” mayinclude a first set of gain values and the “other” may include a secondset of gain values. As shown this individual needs less gain applied inthe lower frequencies when hearing himself speak.

In an embodiment, the gain profiles may be further broken down acrossranges of the audible frequency spectrum. FIG. 5 illustratesfrequency-dependent gain distributions for five different consecutivefrequency ranges, corresponding to 500 Hz, 1 kHz, 2 kHz, 4 kHz and 8kHz. These frequency-dependent gain distributions correlate an outputdecibel level generated for each frequency range depending upon itscorresponding input decibel level. These five frequency-dependent gaindistributions may all be associated with the same gain profile, sinceeach one applies to different frequencies. Thus, a similar but somewhatdifferent set of frequency-dependent gain distributions may be providedfor a different gain profile.

FIG. 6 illustrates a process flow of an embodiment method of controlling300 the output of a hearing aid worn by a wearer. In an embodiment, theoperations of method 300 may be performed by a processor of a hearingaid. In block 310 an input audio signal is received. The input audiosignal may represent sounds heard by the hearing aid wearer. Thosesounds may or may not include sounds generated by the wearer's ownspeech. In block 320 the input audio signal may be filtered to removenoise and/or easily defined segments for which the main gain profilesneed not be applied. In block 330 a facial movement indication may bereceived from a facial movement detector, measured contemporaneouslywith the input audio signal. A received facial movement indication mayinclude an unprocessed signal from the facial movement detector (whichmay be received along with other unprocessed signals) that is laterconfirmed and/or determined (after analysis by a signal processor) torepresent a facial movement pattern associated with a wearer speaking(e.g., muscle activity analyzer 240). The other unprocessed signals maybe ones not determined to be associated with a wearer speaking. Thus,the facial movement detector may detect various movements, not all ofwhich are eventually determined to be associated with the wearerspeaking. In block 340, a determination may be made as to whether thefacial movement indication is a movement match of a first movementpattern associated with the wearer speaking. The first movement patternmay be one of many movement patterns associated with human speech ormore specifically with the speech of the hearing aid wearer.Additionally, the determination as to whether a measured movementpattern is a match may be also or alternatively based on other weareractivity associated with the wearer speaking that is not a pattern ofmovement. Such “other wearer activity” may include natural reactionsthat occur while a person is speaking. For example, biological/chemicalchanges in the wearer's skin or other bio-indicators associated with awearer speaking may be used, particularly those measureable/detectablefrom/through a wearer's skin. If the processor determines that thefacial movement indication is a match to the first movement patternassociated with the wearer speaking (i.e., determination block 340 is“Yes”), a first gain profile may be applied to the contemporaneousportion(s) of the input audio signal. If the processor determines thatthe facial movement indication does not match the first movement patternassociated with the wearer speaking (i.e., determination block 340 is“No”), a second gain profile may be applied. In order to more accuratelyidentify facial movements correlated to the wearer actually speaking, indetermination block 350 a further determination may be made. Inparticular, determination block 350 the hearing aid processor maydetermine whether at least a portion of the input audio signal is anaudio match to a first voice pattern associated with speech generated bythe wearer. If the processor determines that at least a portion of theinput audio signal is an audio match to a first voice pattern associatedwith speech generated by the wearer (i.e., determination block 350 is“Yes”), this may be a confirmation that the first gain profile should beapplied to the contemporaneous portion(s) of the input audio signal (seeblock 370). However, if the processor determines that at least a portionof the input audio signal is not an audio match to a first voice patternassociated with speech generated by the wearer (i.e., determinationblock 350 is “No”), a second gain profile may be applied (see block375).

In an embodiment, the method 300 may optionally further discriminategain profile frequency band distributions based on separate and distinctfrequency band ranges. Thus, in block 360 a first gain profile frequencyband distribution may be determined. For example, the input audio signalmay be divided into multiple frequency bands, such as under 500 Hz, 500Hz to under 1 kHz, 1 kHz to under 2 kHz, 2 kHz to under 4 kHz, and 4 kHzto 8 kHz (as illustrated in FIG. 5). In this way, a firstfrequency-dependent gain distribution may be customized for each of thefrequency bands in the first gain profile frequency band distribution.Similarly, in block 365 a first gain profile frequency band distributionmay be determined. In this way, a second frequency-dependent gaindistribution may be customized for each of the frequency bands in thesecond gain profile frequency band distribution. The second gain profilefrequency band distribution may be similar or the same as the first gainprofile frequency band distribution or it may have a differentdistribution. However, if the frequency band distributiondetermination(s) are not made, the determined gain profile may beapplied in block 370 or block 375, depending upon the gain profile thatis to be applied. Once the determined gain profile is applied in block370 or 375, the mixer may combine the augmented input audio signals inblock 380 and a mixed augmented audio segment may be output in block390.

FIG. 7A illustrates an embodiment hearing aid 101 including a housing103, two microphones 110, 112, an ear hook 115, multiple facial movementdetectors 120 (four are shown) and a speaker 130. The facial movementdetectors 120 may be arranged around the ear hook 115 to ensure contactwith skin covering one or more bony structures and/or facial muscles ofthe wearer, such as around the base of the ear, for detecting thewearer's own speech. FIG. 7B illustrates the hearing aid 101 worn by awearer on the externally protruding parts of the ear 10. When worn bythe hearing aid wearer, a first hearing aid microphone 110 may beconfigured to be facing toward the wearer's mouth, while a secondhearing aid microphone 112 may be facing outwardly. In this way, thefirst microphone 100 may be used primarily to detect the wearer's ownspeech. Such a first microphone 110 may be unidirectional, while thesecond microphone 112 may be omni-directional for collecting a broaderrange of ambient sounds. Also, a differential between the firstmicrophone 110 and the second microphone 112 may be used to determine adominant direction of the ambient noise being picked-up by themicrophones. For example, a first gain profile may be applied inresponse to determining that a threshold portion of the input audiosignal was received from the first microphone, while a second gainprofile may be applied if that threshold portion was not received by thefirst microphone. The threshold portion may be greater than fiftypercent or some other percentage determined to reliably identify adominant direction of the ambient noise being picked-up by themicrophones. The multiple facial movement detectors 120 may provideredundancy in case any one of them fails to detect facial movements. Thehearing aid 101 may further include input mechanisms 151, 153 (in theform of an activation button and/or toggle switch) for manual modeselections, in order to allow the wearer to control which mode is used.In this way, a processor of the hearing aid may receive a selectioninput when an input mechanism 151, 153 is used. In this way, such aselection input may indicate whether a particular gain profile should beapplied or used. For example, the wearer may optionally override thegain profile determinations noted above by pressing the input button151. Additionally, one or both of the input mechanisms 151, 153 maycontrol the setup of the hearing aid, such as when the wearer's ownvoice needs to be recorded for calibration of the above-noted profiles.For example, the input mechanism 153 may be used to start and stop therecording of the wearer's own voice. It should be understood that thehearing aids of the various embodiments may include a power source, suchas batteries.

The hearing aid processor(s) may be configured with processor-executableinstructions to receive inputs from the microphones, facial movementdetectors and input mechanism(s), as well as generate outputs from thespeaker. The sensors, such as microphones, facial movement detectors andinput mechanism(s) may be used as means for receiving signals and/orindications. The processor(s) may be used as means for determiningconditions/triggers, such as whether patterns match or as means forapplying a select gain profile. Also, the speaker and/or relatedhardware may be used as means for outputting. The processor may becoupled to one or more internal memories. Internal memories may bevolatile or non-volatile memories, and may also be secure and/orencrypted memories, or unsecure and/or unencrypted memories, or anycombination thereof. The processor may be any programmablemicroprocessor, microcomputer or multiple processor chip or chips thatcan be configured by software instructions (i.e., applications) toperform a variety of functions, including the functions of the variousaspects described above. Multiple processors may be provided, such asone processor dedicated to one or more functions and another one or moreprocessors dedicated to running other applications/functions. Typically,software applications may be stored in the internal memory before theyare accessed and loaded into the processor. The processor may includeinternal memory sufficient to store the application softwareinstructions. In many devices the internal memory may be a volatile ornonvolatile memory, such as flash memory, or a mixture of both. For thepurposes of this description, a general reference to memory refers tomemory accessible by the processor including internal memory orremovable memory plugged into the hearing aid and memory within theprocessor.

The processors in the various embodiments described herein may be anyprogrammable microprocessor, microcomputer or multiple processor chip orchips that can be configured by software instructions(applications/programs) to perform a variety of functions, including thefunctions of the various embodiments described above. Typically,software applications may be stored in the internal memory before theyare accessed and loaded into the processors. The processors may includeinternal memory sufficient to store the processor-executable softwareinstructions. In many devices the internal memory may be a volatile ornonvolatile memory, such as flash memory, or a mixture of both. For thepurposes of this description, a general reference to memory refers tomemory accessible by the processors including internal memory orremovable memory plugged into the device and memory within the processorthemselves.

In one or more exemplary embodiments, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software, the functions may be stored as one or moreinstructions or code on a non-transitory computer readable storagemedium or non-transitory processor-readable storage medium. The steps ofa method or algorithm may be embodied in a processor-executable softwaremodule which may reside on a non-transitory computer readable orprocessor-readable storage medium. Non-transitory computer readable orprocessor-readable storage media may be any storage media that may beaccessed by a computer or a processor. By way of example but notlimitation, such non-transitory computer readable or processor-readablemedia may include RAM, ROM, EEPROM, FLASH memory, CD-ROM or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium that may be used to store desired programcode in the form of instructions or data structures and that may beaccessed by a computer. Disk and disc, as used herein, includes compactdisc (CD), laser disc, optical disc, digital versatile disc (DVD),floppy disk, and blu-ray disc where disks usually reproduce datamagnetically, while discs reproduce data optically with lasers.Combinations of the above are also included within the scope ofnon-transitory computer readable and processor-readable media.Additionally, the operations of a method or algorithm may reside as oneor any combination or set of codes and/or instructions on anon-transitory processor-readable medium and/or computer readablemedium, which may be incorporated into a computer program product.

The foregoing method descriptions and the process flow diagrams areprovided merely as illustrative examples and are not intended to requireor imply that the blocks of the various embodiments must be performed inthe order presented. As will be appreciated by one of skill in the artthe order of blocks in the foregoing embodiments may be performed in anyorder.

Words such as “thereafter,” “then,” “next,” etc. are not intended tolimit the order of the blocks; these words are simply used to guide thereader through the description of the methods. Further, any reference toclaim elements in the singular, for example, using the articles “a,”“an” or “the” is not to be construed as limiting the element to thesingular. Additionally, as used herein and particularly in the claims,“comprising” has an open-ended meaning, such that one or more additionalunspecified elements, steps and aspects may be further included and/orpresent.

The various illustrative logical blocks, modules, circuits, and processflow diagram blocks described in connection with the embodiments may beimplemented as electronic hardware, computer software, or combinationsof both. To clearly illustrate this interchangeability of hardware andsoftware, various illustrative components, blocks, modules, circuits,and blocks have been described above generally in terms of theirfunctionality. Whether such functionality is implemented as hardware orsoftware depends upon the particular application and design constraintsimposed on the overall system. Skilled artisans may implement thedescribed functionality in varying ways for each particular application,but such implementation decisions should not be interpreted as causing adeparture from the scope of the present invention.

The preceding description of the disclosed embodiments is provided toenable any person skilled in the art to make or use the presentinvention. Various modifications to these embodiments will be readilyapparent to those skilled in the art, and the generic principles definedherein may be applied to other embodiments without departing from thespirit or scope of the invention. Thus, the present invention is notintended to be limited to the embodiments shown herein but is to beaccorded the widest scope consistent with the following claims and theprinciples and novel features disclosed herein.

What is claimed is:
 1. A method of controlling an output of a hearingaid worn by a wearer, comprising: receiving an input audio signal from amicrophone; receiving a facial movement indication from a facialmovement detector measured contemporaneously with the input audiosignal; determining whether the facial movement indication matches afirst movement pattern associated with the wearer speaking; applying afirst gain profile to the input audio signal for generating an augmentedaudio segment in response to determining that the facial movementindication matches the first movement pattern associated with the wearerspeaking; applying a second gain profile to the input audio signal forgenerating the augmented audio segment in response to determining thefacial movement indication does not match the first movement patternassociated with the wearer speaking; and outputting the augmented audiosegment.
 2. The method of claim 1, further comprising: determiningwhether at least a portion of the input audio signal matches a firstvoice pattern associated with speech by the wearer, wherein applying thefirst gain profile is further in response to determining that the inputaudio signal matches the first voice pattern; and applying the secondgain profile to the input audio signal for generating the augmentedaudio segment in response to determining that the input audio signaldoes not match the first voice pattern.
 3. The method of claim 1,wherein the facial movement detector is an electromyography sensor. 4.The method of claim 1, wherein the facial movement detector isconfigured on the hearing aid to engage skin of the wearer.
 5. Themethod of claim 1, wherein the microphone includes a first hearing aidmicrophone and a second hearing aid microphone separate from the firsthearing aid microphone, wherein applying the first gain profile isapplied in response to determining that a threshold portion of the inputaudio signal was received from the first hearing aid microphone.
 6. Themethod of claim 1, wherein the first gain profile applies a first set ofgain values to multiple frequencies of the input audio signal, the firstset of gain values based on a first frequency-dependent gaindistribution of the wearer while the wearer is speaking.
 7. The methodof claim 1, wherein the second gain profile applies a second set of gainvalues to multiple frequencies of the input audio signal, the second setof gain values based on a second frequency-dependent gain distributionof the wearer while the wearer is not speaking.
 8. The method of claim1, further comprising: determining whether the facial movementindication matches other wearer activity associated with the wearerspeaking that is not a pattern of movement, wherein the first gainprofile is applied to the input audio signal for generating theaugmented audio segment in response to determining that the facialmovement indication includes the other wearer activity.
 9. The method ofclaim 1, further comprising: receiving a selection input indicatingwhether the first gain profile should be applied, wherein the first gainprofile is applied to the input audio signal in response to theselection input indicating the first gain profile should be applied andthe second gain profile is applied to the input audio signal in responseto the selection input indicating the first gain profile should not beapplied.
 10. A hearing aid for adjusting sound heard by a wearer,comprising: a facial movement detector configured to detect facialmovements of the wearer; a microphone configured to receive an inputaudio signal in response to detecting sound; a speaker for outputting anaugmented audio segment; a memory having stored thereon first movementpattern data associated with the wearer speaking; and a processorcoupled to the facial movement detector, the microphone, the speaker andthe memory, wherein the processor is configured withprocessor-executable instructions to perform operations comprising:receiving the input audio signal from the microphone; receiving a facialmovement indication from the facial movement detector measuredcontemporaneously with the input audio signal; determining whether thefacial movement indication matches the first movement pattern; applyinga first gain profile to the input audio signal for generating theaugmented audio segment in response to determining that the facialmovement indication matches the first movement pattern; applying asecond gain profile to the input audio signal for generating theaugmented audio segment in response to determining the facial movementindication does not match the first movement pattern; and outputting theaugmented audio segment to the speaker.
 11. The hearing aid of claim 10,wherein the processor is configured with the processor-executableinstructions to perform operations further comprising: determiningwhether at least a portion of the input audio signal matches a firstvoice pattern associated with speech by the wearer, wherein applying thefirst gain profile is further in response to determining that the inputaudio signal matches the first voice pattern; and applying the secondgain profile to the input audio signal for generating the augmentedaudio segment in response to determining that the input audio signaldoes not match the first voice pattern.
 12. The hearing aid of claim 10,wherein the processor is configured with the processor-executableinstructions to perform operations such that the facial movementdetector is an electromyography sensor.
 13. The hearing aid of claim 10,wherein the processor is configured with the processor-executableinstructions to perform operations such that the facial movementdetector is configured on the hearing aid to engage skin of the wearer.14. The hearing aid of claim 10, wherein the processor is configuredwith the processor-executable instructions to perform operations suchthat the microphone includes a first hearing aid microphone and a secondhearing aid microphone separate from the first hearing aid microphone,wherein applying the first gain profile is applied in response todetermining that a threshold portion of the input audio signal wasreceived from the first hearing aid microphone.
 15. The hearing aid ofclaim 10, wherein the processor is configured with theprocessor-executable instructions to perform operations such that thefirst gain profile applies a first set of gain values to multiplefrequencies of the input audio signal, the first set of gain valuesbased on a first frequency-dependent gain distribution of the wearerwhile the wearer is speaking.
 16. The hearing aid of claim 10, whereinthe processor is configured with the processor-executable instructionsto perform operations such that the second gain profile applies a secondset of gain values to multiple frequencies of the input audio signal,the second set of gain values based on a second frequency-dependent gaindistribution of the wearer while the wearer is not speaking.
 17. Thehearing aid of claim 10, wherein the processor is configured with theprocessor-executable instructions to perform operations furthercomprising: determining whether the facial movement indication matchesother wearer activity associated with the wearer speaking that is not apattern of movement, wherein the first gain profile is applied to theinput audio signal for generating the augmented audio segment inresponse to determining that the facial movement indication includes theother wearer activity.
 18. The hearing aid of claim 10, wherein theprocessor is configured with the processor-executable instructions toperform operations further comprising: receiving a selection inputindicating whether the first gain profile should be applied, wherein thefirst gain profile is applied to the input audio signal in response tothe selection input indicating the first gain profile should be appliedand the second gain profile is applied to the input audio signal inresponse to the selection input indicating the first gain profile shouldnot be applied.
 19. A hearing aid for adjusting sound heard by a wearer,comprising: means for receiving an input audio signal from a microphone;means for receiving a facial movement indication from a facial movementdetector measured contemporaneously with the input audio signal; meansfor determining whether the facial movement indication matches a firstmovement pattern associated with the wearer speaking; means for applyinga first gain profile to the input audio signal for generating anaugmented audio segment in response to determining that the facialmovement indication matches the first movement pattern associated withthe wearer speaking; means for applying a second gain profile to theinput audio signal for generating the augmented audio segment inresponse to determining the facial movement indication does not matchthe first movement pattern associated with the wearer speaking; andmeans for outputting the augmented audio segment.
 20. The hearing aid ofclaim 19, further comprising: means for determining whether at least aportion of the input audio signal matches a first voice patternassociated with speech by the wearer, wherein applying the first gainprofile is further in response to determining that the input audiosignal matches the first voice pattern; and means for applying thesecond gain profile to the input audio signal for generating theaugmented audio segment in response to determining that the input audiosignal does not match the first voice pattern.
 21. The hearing aid ofclaim 19, wherein the facial movement detector is an electromyographysensor.
 22. The hearing aid of claim 19, wherein the facial movementdetector is configured on the hearing aid to engage skin of the wearer.23. The hearing aid of claim 19, wherein the microphone includes a firsthearing aid microphone and a second hearing aid microphone separate fromthe first hearing aid microphone, wherein applying the first gainprofile is applied in response to determining that a threshold portionof the input audio signal was received from the first hearing aidmicrophone.
 24. The hearing aid of claim 19, wherein the first gainprofile applies a first set of gain values to multiple frequencies ofthe input audio signal, the first set of gain values based on a firstfrequency-dependent gain distribution of the wearer while the wearer isspeaking.
 25. The hearing aid of claim 19, wherein the second gainprofile applies a second set of gain values to multiple frequencies ofthe input audio signal, the second set of gain values based on a secondfrequency-dependent gain distribution of the wearer while the wearer isnot speaking.
 26. The hearing aid of claim 19, further comprising: meansfor determining whether the facial movement indication matches otherwearer activity associated with the wearer speaking that is not apattern of movement, wherein the first gain profile is applied to theinput audio signal for generating the augmented audio segment inresponse to determining that the facial movement indication includes theother wearer activity.
 27. The hearing aid of claim 19, furthercomprising: means for receiving a selection input indicating whether thefirst gain profile should be applied, wherein the first gain profile isapplied to the input audio signal in response to the selection inputindicating the first gain profile should be applied and the second gainprofile is applied to the input audio signal in response to theselection input indicating the first gain profile should not be applied.28. A non-transitory computer readable storage medium having storedthereon processor-executable instructions configured to cause aprocessor of a hearing aid worn by a wearer to perform operations forcontrolling an output of the hearing aid, the operations comprising:receiving an input audio signal from a microphone; receiving a facialmovement indication from a facial movement detector measuredcontemporaneously with the input audio signal; determining whether thefacial movement indication matches a first movement pattern associatedwith the wearer speaking; applying a first gain profile to the inputaudio signal for generating an augmented audio segment in response todetermining that the facial movement indication matches the firstmovement pattern associated with the wearer speaking; applying a secondgain profile to the input audio signal for generating the augmentedaudio segment in response to determining the facial movement indicationdoes not match the first movement pattern associated with the wearerspeaking; and outputting the augmented audio segment.
 29. Thenon-transitory computer readable storage medium of claim 28, wherein theprocessor-executable instructions are configured to cause the processorof the hearing aid to perform operations further comprising: determiningwhether at least a portion of the input audio signal matches a firstvoice pattern associated with speech by the wearer, wherein applying thefirst gain profile is further in response to determining that the inputaudio signal matches the first voice pattern; and applying the secondgain profile to the input audio signal for generating the augmentedaudio segment in response to determining that the input audio signaldoes not match the first voice pattern.
 30. The non-transitory computerreadable storage medium of claim 28, wherein the processor-executableinstructions are configured to cause the processor of the hearing aid toperform operations such that the facial movement detector is anelectromyography sensor.
 31. The non-transitory computer readablestorage medium of claim 28, wherein the processor-executableinstructions are configured to cause the processor of the hearing aid toperform operations such that the facial movement detector is configuredon the hearing aid to engage skin of the wearer.
 32. The non-transitorycomputer readable storage medium of claim 28, wherein theprocessor-executable instructions are configured to cause the processorof the hearing aid to perform operations such that the microphoneincludes a first hearing aid microphone and a second hearing aidmicrophone separate from the first hearing aid microphone, whereinapplying the first gain profile is applied in response to determiningthat a threshold portion of the input audio signal was received from thefirst hearing aid microphone.
 33. The non-transitory computer readablestorage medium of claim 28, wherein the processor-executableinstructions are configured to cause the processor of the hearing aid toperform operations such that the first gain profile applies a first setof gain values to multiple frequencies of the input audio signal, thefirst set of gain values based on a first frequency-dependent gaindistribution of the wearer while the wearer is speaking.
 34. Thenon-transitory computer readable storage medium of claim 28, wherein theprocessor-executable instructions are configured to cause the processorof the hearing aid to perform operations such that the second gainprofile applies a second set of gain values to multiple frequencies ofthe input audio signal, the second set of gain values based on a secondfrequency-dependent gain distribution of the wearer while the wearer isnot speaking.
 35. The non-transitory computer readable storage medium ofclaim 28, wherein the processor-executable instructions are configuredto cause the processor of the hearing aid to perform operations furthercomprising: determining whether the facial movement indication matchesother wearer activity associated with the wearer speaking that is not apattern of movement, wherein the first gain profile is applied to theinput audio signal for generating the augmented audio segment inresponse to determining that the facial movement indication includes theother wearer activity.
 36. The non-transitory computer readable storagemedium of claim 28, wherein the processor-executable instructions areconfigured to cause the processor of the hearing aid to performoperations further comprising: receiving a selection input indicatingwhether the first gain profile should be applied, wherein the first gainprofile is applied to the input audio signal in response to theselection input indicating the first gain profile should be applied andthe second gain profile is applied to the input audio signal in responseto the selection input indicating the first gain profile should not beapplied.